Logistic regression cost function

Author: Leonardo Espin

Date: 12/8/2018

Using python's symbolic math library to calculate the partial derivatives of the logistic cost function, which are used in gradient descent calculations for finding the optimal parameters of the regression

In [1]:
#symbolic computing library in python
from sympy import *
init_printing() #To get nice-looking  formatted output run

In SymPy we need to create symbols for the variables we want to work with:

In [2]:
z,y,s = symbols('z,y,s')
x1, x2, x3 = symbols("x1, x2, x3")
T0,T1, T2, T3 = symbols("T0,T1, T2, T3") #thetas
In [3]:
h = T0+T1*x1+T2*x2+T3*x3
In [4]:
g = 1/(1+exp(-z)) #the sigmoid function
In [5]:
f=g.subs({z:h}) #the hypothesis
f
Out[5]:
$$\frac{1}{e^{- T_{0} - T_{1} x_{1} - T_{2} x_{2} - T_{3} x_{3}} + 1}$$
In [6]:
COST=-y*log(s) -(1-y)*log(1-s)

the logistic cost function for a single observation:

In [7]:
J=COST.subs({s:f}) 
J
Out[7]:
$$- y \log{\left (\frac{1}{e^{- T_{0} - T_{1} x_{1} - T_{2} x_{2} - T_{3} x_{3}} + 1} \right )} - \left(- y + 1\right) \log{\left (1 - \frac{1}{e^{- T_{0} - T_{1} x_{1} - T_{2} x_{2} - T_{3} x_{3}} + 1} \right )}$$

partial derivative with respect to one of the parameters $\partial J/\partial \theta_1$:

In [8]:
simplify(diff(J, T1))
Out[8]:
$$\frac{x_{1} \left(- y e^{- T_{0} - T_{1} x_{1} - T_{2} x_{2} - T_{3} x_{3}} - y + 1\right)}{e^{- T_{0} - T_{1} x_{1} - T_{2} x_{2} - T_{3} x_{3}} + 1}$$

which is the same as:

$$\left( \frac{1-y\left(1+e^{-T_0 -T_1 x_1 -T_2 x_2 -T_3 x_3}\right)}{1+e^{-T_0 -T_1 x_1 -T_2 x_2 -T_3 x_3}}\right)x_1$$

or

$$ (h_\theta (x) -y)x_1,\quad x=(x_1,x_2,x_3)$$